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about me ... 

» Studied Media Management with an empirical focus 

» 2010-2013 Games Analyst, Lead Games Analyst and Head of 

Analytics at Bigpoint GmbH 

» Since 2013 Head of Analytics at InnoGames GmbH 
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about InnoGames ... 

» Started 2003 as a Hobby-project from our Managing Directors, 
today the classic Tribal Wars is still growing after 11 years 

» F2P-Business-model, started on browser, transition to cross- 
platform developer & publisher 

» 330 employees from 25 nations, 400 until the end of the year 

» 130 Mio. registered players, ~70 million € in revenues in 2013 

(and growing) 



Tribal Wars 2 
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about the team ... 



churn-reason 

survey 


churn-patterns & 
churn-user scoring 


support buildup of 
new data-sets, merge 
complex data sets 


define KPIs, 
benchmark 
performance 


consult product 
owners 


analytics 


cluster user- 
segments, predict 
user-actions 


A *>»- ■ 


define & analyze A/B- 
or multivariate tests 


HOW 


maiket 

research 


WHY 


external market 
data, market- 
conditions 


perform user tests to 
explore appeal of 
products 


user- 

potential & 
churn- 

development 


inform relevant 
stakeholders, 
identify risks and 
potentials 


conduct surveys to 
understand user- 
motives & needs 


differentiate target-groups 





2 examples of valuable standard-reporting 
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spreadsheets are not all that bad ... 


1. 

2 . 


tutorial-conversion-funnel 

reporting 

-> 75% higher drop out for FoE in 
first quest due to synced bug 


event evaluations 

-> constant improvement of 
participation rates due to 
awareness and detail- 
optimization 





drop out for first tutorial quest 


21% 



participation rates for major 
Grepolis-events 


76% 


60% 


I 


Halloween 


38% 




Christmas 


80% 


67% 


Easter 


■ Browser ■ Mobile 




in-house Event Tracking 
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still, there's more data ... 

backend 



Stage 1 Stage 2 Stage 3 Stage 4 






all stages scale out and use open -source-software 


open source 
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too good just for fun facts 


60 GB 


new event-data every day 
saved on a Hadoop-cluster with currently 44 servers 


> 400 M 


logged events every day 


every day ... 

1.1 Mio. hours of InnoGames are played all around the globe 
10 Mio. neighbors are visited in Forge of Empires 
74 Mio. units are recruited in Grepolis 




churn reasons 
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churn: the hard truth about free-to-play 


spreading loss is high: cohort-view 

100% 



registrations ... on the ... in the ... after 3 ... after 6 ... after one 

second day second months months year 

month 


Naturally, most churn happens 
whatever you do ... 


29% 


32% 


18% 


22 % 



l I 


other things now prefer playing game is too time- something in the 
more important another game consuming game has made 
than games me quit 
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a stable 


easy to 
achieve, 
but 

expensive 


k 


mostly a 
constant 


hard to 
change, 
but a 
valuable 
trigger 


user-basis requires constant input: 
monthly view 



2014-01 2014-02 2014-03 2014-04 

Churn ■ Reactivation «New 



nhappy with the 
game 


was conquered/attacked too often, 
alliance disbanded 

boredom, perceived unfairness, not 
enough freedom, too simple, too complex 


... but there is enough potential 
where incentives & 
communication might help 











GAME DEVELOPERS CONFERENCE’” EUROPE 2014 AUGUST 11-13, 2014 


GDCEUROPE.C0M 


first steps in prediction modelling 


GREF0LIS 



predict churn of mid-game players at a time where they are still active 


43 defined parameters, e.g. playtime, deff- 
battles, off-battles, completed quests ... 


REGISTRATION 


ACTIVITY 


PREDICTION 




day 0 


day 6 


Y 

Prediction input is based on all events 
within the first 4 weeks after registration 


day 27 


day 35 


day 48 

J 


Y 

Goal: Predicting if player will 
churn or stay active in week 6 + 7 



decision tree models 

Classification of objects based 
on decision-rules until , optimal' 
classification is reached 


R party-package 

works with all types of 
data, controls for 
overfitting 



benchmark of different 
parameter-sets to find a 
sweet-spot between active- 
and churn-prediction 
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actually quite simple: a decision tree model 
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GREF0LIS 
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actually quite simple: a decision tree model 


GREF0LIS 



>0.58 


all users with a playtime of less than half an 
hour in the fourth week are more likely to 
churn than to remain active (score above 0.5) 


when a town was lost or the player is not in 
an ally the churn-probability rises 
additionally. Having paid, a higher points- 
progress or login-count lowers the churn- 
probability 


<1.09 


<0 >0 


<7.47 


>7.47 


/ <800 / \>800 \ 

/ / \ \ 


> 0 


<11.78/ >n . 78 


<0 >0 


1 ■ r 


> i 


<3.4 / > 3.4 


0,8 0,9 0,7 0,9 0,8 0,6 
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actually quite simple: a decision tree model 


GREF0LIS 


<0.26 


exampe rule: players with less than 15 
minutes playtime in the 4th week (but more 
than 2 minutes), who did not lose a town so 
far and logged in more than 17 times in total 


<o 


> i 


>o 


>0 


<0 


<17 >17 


>0 


<794 >794 



70% of these players are churning 


> 20 


> 0 


<11.78/ >n . 78 


<3.4 / > 3.4 


-/ 
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actually quite simple: a decision tree model 


GREF0LIS 


<o 


>0.58 


<0.58 


<0.26 


if users play more than half an hour in the 
fourth week, but lose their last or only town, 
the churn probability is also very high 


>0.04 




>0 


<0 


>0 


for some users the town loss leads to a strong 
counter-reaction: a new world account and a 
high playtime (more than 3 hours in the 4th 
week). In this case, the churn-probability 
decreases again to a low level 




l 

> 17 



<1648 / 

/ 
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A 

\^>1.09 


A 
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prediction power 
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first test: nice start ... 


improved timing/ no cohort-approach 


ACTIVITY M & t LAYER G PREDICT I° N 


+ 


+ 


-I— ► 


week 0 


week 1 


week 2 


reality 


► 


C 

o 


T3 

CD 



no churn 

churn 

no churn 

95% 

43% 

churn 

5% 

57% 


r- 

odel-hitra 

80% 


L 


A 



Layer + 1,000 Bonds 
as incentive 


no action 


incentive value: 

~20 € 


I M hrmpr <* irt/1 «» MHufcumu' I 


random 

assignment 


Kinc Vuhrkhl fur <lich! 


Howdy Mobuiaol 


t>» SfcMV 'IcWrt 6* MiM Giwl» tun itna 
uW-» rht Or all Anaiturrung dana r Titan 100 
Booth 0«»« fcanntt du m Unwn-P«ofc-Shcp 
o*'>-n tau*ctMd 


VWI Spafi damt wurracM 
Don T>>« Wail Ttitvt 


Avg daily retention in 
week 2 

21,2% 



control group test-group 


test ) model 
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but a lot potential 


100k users 


(fortunately) 

non-churners 

dominate 



active 


churn 


active 

churn | 

95% 

43% 

5% 

57% 


calculated 

with 

equally 

divided 

sample 




active 

churn j 

BSBfl 

86k 

- 4k 

j churn 

4k 

6k 


44% falsely 
targeted 


model 


GDCEUR0PE.C0M 



Sheriff 


seems to be 

CD 

rather 

>* 

sceptical 

4 -> 

to 

that you get 


a reward ... 



g \ 


only 63% of the users who 
received the incentive cashed 
the reward afterwards -> user- 
flow (inventory) was unclear for 
churn-candidates 


v J 
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second test 






r 

model- 

1 

L < 

hitrate 

n°A 

3 1 

L 


J 



32% wrongly targeted 

(42% for first test) 

(but a lot of lost potential - see 
false positives) 



Avg daily Retention in week 2 



to 

CD 


■ Test Group 1 (Layer + 1,000 Bonds) 

■ Test Group 2 (Layer + eye-catcher + 1,000 Bonds) 

¥ 










new approach 
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model optimization 


o 


Aggregated user-states over a certain-time frame 


daily playtime [in hours] of player A 


daily playtime [in hours] of player B 




playing time last 3 weeks: 

35 hours 


playing time last 3 weeks: 

36 hours 


old approach would consider both users as almost identical; changes in 
behavior/ playing trend are ignored 




Integrate variables for every user which reflect activity development 


Churn = no login in a 
particular time-period 


6 


softer churn-criteria 


not " the player has 
left the game" but 
"the player's activity 
has dropped below 
the churn threshold" 

Threshold: not more 
than 3 days with 
game logins during 
the last 30 calendar 
days 


credit to: Dmitry Nozhin [http://www.gamasutra.com/view/feature/176747/predicting_churn_when_do_veterans_.php] 








days of play in last 30 calendar days 
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model optimization 





4-9 



0-3 



churn-probability 


-> 0 % 
♦ 11 % 
> 32 % 


> 47 % 


*■ 66 % 


* 91 % 
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hitrate 


100 % 


97 % 


93 % 


92 % 


all activity 
segments are 
predicted with 
comparable 
accuracy 


91% 


94 % 
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third test 


improved model/ scoring: 
trend variables, churn-threshold 
instead of churn 


improved layer apperance 


1 


A/B Test of incentive: 
premium currency vs. item 





active 

churn 


96% 

6% 


4% 

94% 



r 

model- 

1 

L < 

hitrate 

)5°/< 

3 i 
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<L> 

TD 

O 

E 


12% wrongly targeted 

(32% for second test) 



Avg. daily retention in week 4 

14% 


13% 



c n 

Q) 


Control-Group (no action) 
l Test Group 1 (layer + 1,000 Bonds) 
I Test Group 2 (layer + item) 
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more positive details 


the stricter the retention- 
criteria, the higher the impact 

unique retention-plus 

(compared to control-group) 


more than 1 
login 


more than 2 
logins 


more than 3 
logins 


I more than 4 
I logins 

V _ 


9% 


4% 



17% 


17% 


6% 



layer + bonds ■ layer + items 


surprisingly, the impact is 
sustainable 

test 2: effect in week 2 

18% 

8 % 



Layer + 1,000 Bonds Layer + eye-catcher + 
1,000 Bonds 


test 2: effect in week 6 


13% 


iu% 

■ l 

Layer + 1,000 Bonds Layer + eye-catcher + 
1,000 Bonds 


even though not the target, 
there is evidence for a 
monetization-effect 

test 2: lifetime-value per test- 
group 


131% 


100 % 


110 % 


Control-group Layer + 1,000 Layer + eye- 
Bonds catcher + 1,000 
Bonds 


numbers should be taken with a grain 
of salt as variance for revenue 
numbers is high. 
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roll-out 




incentive value ~20 EUR 


incentive value ~2 EUR 


r 

no 

1 

r 

sig 
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L 

effect 

J 
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learnings 


» In general: Don't wait for a perfect all-around solution, just start testing with the 
existing data and possibilities 

» For event-tracking: Every additional bit of information helps - track the most 

important events and expand from there, do not start million C-projects that will deliver 
results in years 

» Also for event-tracking: take your time for data-QA - the first implementation will 
have many flaws. Play the game yourself and monitor the results 

» For modelling: Include behavior changes and playing trends as input for your models/ 
target churn threshold instead of 0 logins. Think about the right timing for the process 

» For messaging: Little variations can have a huge effect - test every element of the 
interstitial 

» For incentives: The incentive should give the user enough freedom so that he starts 
exploring the possibilities of your game 
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outlook/ next steps 

automatization of scoring v 

central CRM-solution with targeting 
logic and campaign-tracking 



C/5 

C/5 

<L> 

U 

O 


further tests of timing, content and incentive 


individual sales/ pay-reactivation 


individual game help and feature triggering 


content 
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outlook/ social network analysis 


communication-ties 
between users of one 
world 


\ * 
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thanks to ... 

... the analytics team, especially Christoph Scholtysik, 

for the great work on this and every other project 
... the CRM team, especially Thomas Cartwright, for the 
excellent collaboration 

... the game teams for implementing all this stuff 

and thanks for your attention! 


Michael Lenz 
Head of Analytics 
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FriesenstraBe 13 - 20097 Hamburg - Germany 

michaeLlenz@innoqames.com 



Business booth: Hall 4.2, D53 
Entertainment booth: Hall 10.1, C15 



GDCEUROPE.COM 


^InnoGames 


corporate.innogames.com 




